Frame interpolation creates an image frame from neighboring images. The neighboring images may be fields in an interlaced video format, used to form a frame of data, or neighboring frames of a soon-to-be-created frame. Typically, frame interpolation has the goal of increasing the number of frames. For example, one may desire to convert from a 24 frame per second rate, such as in film, to 120 frames per second for a video display device such as a liquid crystal display.
In the simplest approach, one could increase the frame rate by repeating the most recent frame until the next frame is ready for display. However, this does not account for moving objects which may appear to jump from frame to frame and have flickering artifacts. Motion estimation and motion compensation techniques may alleviate some of these issues. These techniques rely upon motion vectors to shift the image data for the moving object to the correct position in interpolated frames, thereby compensating for the motion of the object.
Therefore, true motion estimation techniques are very important in frame interpolation. Accurate motion estimation generates high quality frame interpolation results. Considering the computational cost for real time application, most motion estimation algorithms are block based motion estimation that uses block level SAD (sum of absolute differences) value and spatial and temporal smoothness constraint as the criteria. However, the block level motion estimation cannot deal with object level motion and non-translational motion well. To solve these two problems, different solutions have been proposed.
One possible solution is to use a segmentation method, typically used to generate object level motion. For example, if the images contain a car, accurate segmentation would result in the car body segmented from the wheel. The motion vectors would then define a motion model for each segment. Segmentation is expensive from a computational standpoint and the expense is increased by the need to find the correct motion model for each segment. The motion model for the car body would be one type, such as translational, and the motion model for the wheel would be a more complex model, such as affine.
A low cost approach that integrates both block and local motion would be helpful.